Search

Characterizing and predicting the training performance of modern machine learning (ML) workloads on compute systems with compute and communication spread between CPUs, GPUs, and network devices is not only the key to optimization and planning but also a complex goal to achieve. The primary challenges include the complexity of synchronization and load balancing between CPUs and GPUs, the variance in input data distribution, and the use of different communication devices and topologies (e.g., NVLink, PCIe, network cards) that connect multiple compute devices, coupled with the desire for flexible training configurations. Built on top of our prior work for single-GPU platforms, we address these challenges and enable multi-GPU performance modeling by incorporating (1) data-distribution-aware performance models for embedding table lookup, and (2) data movement prediction of communication collectives, into our upgraded performance modeling pipeline equipped with inter-and intra-rank synchronization for ML workloads trained on multi-GPU platforms. Beyond accurately predicting the per-iteration training time of DLRM models with random configurations with a geomean error of 5.21% on two multi-GPU platforms, our prediction pipeline generalizes well to other types of ML workloads, such as Transformer-based NLP models with a geomean error of 3.00%. Moreover, even without actually running ML workloads like DLRMs on the hardware, it is capable of generating insights such as quickly selecting the fastest embedding table sharding configuration (with a success rate of 85%)., Comment: 12 pages, 11 figures, 4 tables

Type IIn supernovae (SNe) exhibit narrow hydrogen lines that arise from the strong interaction between ejecta and circumstellar material. It remains poorly understood, however, what progenitor stars give rise to these explosions. In this work, we perform a detailed analysis of the progenitor and environment of the nearby Type IIn SN 2010jl. With newer images taken by the Hubble Space Telescope, we confirm that the previously reported progenitor candidate is a blend of the progenitor itself and a field star cluster in its close vicinity. SN 2010jl has now become much fainter than the progenitor. The progenitor is very blue and luminous with an effective temperature of log $T_{\rm eff}/{\rm K}$=4.26$^{+0.11}_{-0.09}$ and a luminosity of log $L/L_{\odot}$ =6.52$^{+0.20}_{-0.16}$. It is located in a very young star-forming region, but its luminosity is much higher than that expected from the environmental stellar populations. We suggest that the progenitor was in outburst when observed. Its nature and evolutionary history remain to be investigated., Comment: ApJL Accepted; 4 figures; comments are welcome

Large language models (LLMs) such as GPT-3, OPT, and LLaMA have demonstrated remarkable accuracy in a wide range of tasks. However, training these models can incur significant expenses, often requiring tens of thousands of GPUs for months of continuous operation. Typically, this training is carried out in specialized GPU clusters equipped with homogeneous high-speed Remote Direct Memory Access (RDMA) network interface cards (NICs). The acquisition and maintenance of such dedicated clusters is challenging. Current LLM training frameworks, like Megatron-LM and Megatron-DeepSpeed, focus primarily on optimizing training within homogeneous cluster settings. In this paper, we introduce Holmes, a training framework for LLMs that employs thoughtfully crafted data and model parallelism strategies over the heterogeneous NIC environment. Our primary technical contribution lies in a novel scheduling method that intelligently allocates distinct computational tasklets in LLM training to specific groups of GPU devices based on the characteristics of their connected NICs. Furthermore, our proposed framework, utilizing pipeline parallel techniques, demonstrates scalability to multiple GPU clusters, even in scenarios without high-speed interconnects between nodes in distinct clusters. We conducted comprehensive experiments that involved various scenarios in the heterogeneous NIC environment. In most cases, our framework achieves performance levels close to those achievable with homogeneous RDMA-capable networks (InfiniBand or RoCE), significantly exceeding training efficiency within the pure Ethernet environment. Additionally, we verified that our framework outperforms other mainstream LLM frameworks under heterogeneous NIC environment in terms of training efficiency and can be seamlessly integrated with them., Comment: 12 pages

In this paper, we study effects of permanent charges on ion flows through membrane channels via a quasi-one-dimensional classical Poisson-Nernst-Planck system. This system includes three ion species, two cations with different valences and one anion, and permanent charges with a simple structure, zeros at the two end regions and a constant over the middle region. For small permanent charges, our main goal is to analyze the effects of permanent charges on ionic flows, interacting with the boundary conditions and channel structure. Continuing from a previous work, we investigate the problem for a new case toward a more comprehensive understanding about effects of permanent charges on ionic fluxes.

Recent advancements in unsupervised protein language models (ProteinLMs), like ESM-1b and ESM-2, have shown promise in different protein prediction tasks. However, these models face challenges due to their high computational demands, significant memory needs, and latency, restricting their usage on devices with limited resources. To tackle this, we explore post-training quantization (PTQ) for ProteinLMs, focusing on ESMFold, a simplified version of AlphaFold based on ESM-2 ProteinLM. Our study is the first attempt to quantize all weights and activations of ProteinLMs. We observed that the typical uniform quantization method performs poorly on ESMFold, causing a significant drop in TM-Score when using 8-bit quantization. We conducted extensive quantization experiments, uncovering unique challenges associated with ESMFold, particularly highly asymmetric activation ranges before Layer Normalization, making representation difficult using low-bit fixed-point formats. To address these challenges, we propose a new PTQ method for ProteinLMs, utilizing piecewise linear quantization for asymmetric activation values to ensure accurate approximation. We demonstrated the effectiveness of our method in protein structure prediction tasks, demonstrating that ESMFold can be accurately quantized to low-bit widths without compromising accuracy. Additionally, we applied our method to the contact prediction task, showcasing its versatility. In summary, our study introduces an innovative PTQ method for ProteinLMs, addressing specific quantization challenges and potentially leading to the development of more efficient ProteinLMs with significant implications for various protein-related applications., Comment: 8 pages, 4 figures

We develop an NLP-based procedure for detecting systematic nonmeritorious consumer complaints, simply called systematic anomalies, among complaint narratives. While classification algorithms are used to detect pronounced anomalies, in the case of smaller and frequent systematic anomalies, the algorithms may falter due to a variety of reasons, including technical ones as well as natural limitations of human analysts. Therefore, as the next step after classification, we convert the complaint narratives into quantitative data, which are then analyzed using an algorithm for detecting systematic anomalies. We illustrate the entire procedure using complaint narratives from the Consumer Complaint Database of the Consumer Financial Protection Bureau.

As one of the closest supernovae (SNe) in the last decade, SN 2023ixf is an unprecedented target to investigate the progenitor star that exploded. However, there is still significant uncertainty in the reported progenitor properties. In this work, we present a detailed study of the progenitor of SN 2023ixf with two independent analyses. We first modelled its spectral energy distribution (SED) based on Hubble Space Telescope optical, Spitzer mid-infrared (IR), and ground-based near-IR data. We find that stellar pulsation and circumstellar extinction have great impacts on SED fitting, and the result suggests a relatively massive red supergiant (RSG) surrounded by C-rich dust with an initial mass of 16.2--17.4 Msun. The corresponding rate of mass-loss occurring at least 3 years before the SN explosion is about $2 \times 10^{-4} M_\odot$yr$^{-1}$. We also derived the star formation history of the SN environment based on resolved stellar populations, and the most recent star-forming epoch corresponds to a progenitor initial mass of 17--19 Msun, in agreement with that from our SED fitting. Therefore, we conclude that the progenitor of SN 2023ixf is close to the high-mass end for Type II SN progenitors., Comment: 12 pages, 5 figures, accepted by ApJL

Utilizing infrastructure and vehicle-side information to track and forecast the behaviors of surrounding traffic participants can significantly improve decision-making and safety in autonomous driving. However, the lack of real-world sequential datasets limits research in this area. To address this issue, we introduce V2X-Seq, the first large-scale sequential V2X dataset, which includes data frames, trajectories, vector maps, and traffic lights captured from natural scenery. V2X-Seq comprises two parts: the sequential perception dataset, which includes more than 15,000 frames captured from 95 scenarios, and the trajectory forecasting dataset, which contains about 80,000 infrastructure-view scenarios, 80,000 vehicle-view scenarios, and 50,000 cooperative-view scenarios captured from 28 intersections' areas, covering 672 hours of data. Based on V2X-Seq, we introduce three new tasks for vehicle-infrastructure cooperative (VIC) autonomous driving: VIC3D Tracking, Online-VIC Forecasting, and Offline-VIC Forecasting. We also provide benchmarks for the introduced tasks. Find data, code, and more up-to-date information at \href{https://github.com/AIR-THU/DAIR-V2X-Seq}{https://github.com/AIR-THU/DAIR-V2X-Seq}., Comment: CVPR2023